usb: device_next: new USB Video Class (UVC) implementation #76798

josuah · 2024-08-07T14:08:09Z

Dependencies:

~~drivers: video: introduce "GET" sub-operations #78603~~ replaced by something else to come
~~drivers: video: allow allocation with a header preceding the buffer #79168~~ not required anymore
~~video: emul: virtual driver for an imager and RX engine #79482~~
drivers: video: Migrate ov7670 and smartdma to use video interfaces binding #88323
drivers: video: CONFIG_VIDEO_SW_GENERATOR -> zephyr,video-sw-generator #88780
drivers: video: controls: add the BASE and CAMERA controls #88924

Downstream:

drivers: video: sw_pipeline: glue lib/pixel and drivers/video #88839

Build and run:

usb: device_next: new USB Video Class (UVC) implementation #76798 (comment)

Documentation preview:

https://builds.zephyrproject.io/zephyr/pr/76798/docs/samples/subsys/usb/uvc/README.html

Try it on your own branch:

git remote add tvai https://github.com/tinyvision-ai-inc/zephyr
git fetch tvai
git cherry-pick tvai/pr-usb-uvc~1..tvai/pr-usb-uvc~0
west build -b <board> samples/subsys/usb/uvc/

josuah · 2024-08-07T14:09:22Z

I am grateful for the work on the USB and Video stacks of Zephyr, as well as the entire Zephyr tree, on the shoulder of which this is built.

josuah · 2024-08-08T02:16:03Z

Force push:

fixes to make it compile with west build -b frdm_mcxn947/mcxn947/cpu0 (a board I happened to have, open to suggestions)
added another commit (with I hope proper attributions!) to introduce fragments, which UVC and this implementation support.

josuah · 2024-08-08T02:22:20Z

Force-push:

Add a sample to illustrate the usage. It is not yet tested but compiles.

XenuIsWatching · 2024-08-08T18:56:44Z

subsys/usb/device_next/class/usbd_uvc.c

+		.dwMinBitRate = sys_cpu_to_le32(15360000),                                         \
+		.dwMaxBitRate = sys_cpu_to_le32(15360000),                                         \


Where are these values coming from?

These are chosen arbitrarily, and I need to think about what should I do here: pick reasonable defaults? Deduce from other values? Let the user figure out by exposing a devicetree option?

In future work, it will be possible to ask the sensor for VIDEO_CID_PIXEL_RATE in combination with video_bits_per_pixel() to generate this field.

XenuIsWatching · 2024-08-08T18:57:48Z

subsys/usb/device_next/class/usbd_uvc.c

+				.bLength = sizeof(struct usb_ep_descriptor),                       \
+				.bDescriptorType = USB_DESC_ENDPOINT,                              \
+				.bEndpointAddress = 0x81,                                          \
+				.bmAttributes = USB_EP_TYPE_BULK,                                  \


I believe UVC can be BULK or ISO.... but I only see BULK supported here. Do you plan to have a way to select which type of EP?

I forgot to mention that only BULK is supported through this early version. I will need to spend more time investigating the device_next APIs for how ISO is implemented.

I cannot guarantee ETAs as I would do this feature on free time, but will add it to the roadmap: this will be needed at some point.

include/zephyr/usb/class/usb_uvc.h

josuah · 2024-08-08T22:10:23Z

Very grateful for your reviews @XenuIsWatching I will force-push with the changes as soon as I get a chance to do so.

josuah · 2024-08-21T11:35:12Z

Force-push:

control a selected source = <&dev>; device from the UVC stack directly
wait that a format got selected before queuing video buffers to the USB
use "discrete" frame intervals (FPS) values, which seemed to help getting it to work under Windows
USB3CV compliant descriptors

josuah · 2024-09-07T15:47:51Z

force push: rebased on main

josuah · 2024-09-07T15:56:50Z

force push: changed the way descriptors are declared and introduce video controls at the descriptor level (no support for controls commands yet)

josuah · 2024-09-12T00:44:11Z

force push: implemented the UVC controls for the supported Video class controls.

I did not test this yet with the samples, but on the internal fork, we could get an IMX219 sensor with exposure and gain control from the host, format selection at startup (but not runtime, see [1]).

I believe that the last step for me is to test the sample on several boards that support device_next and fix the CI.

josuah · 2024-09-12T01:50:49Z

Known limitations:

[1]: there is currently no <video.h> API to let the application know that a format change occurred. For now, the sample waits that the host makes an initial choice, and does not reconsider its format selection.
A trade-off between supported feature and sample complexity.

[2]: there is currently no <video.h> API to let the video device communicate their min/max/step, so 0 is always picked as min, INT32/16/8_MAX as max, and 1 as step. Maybe the solution is to define an unit for each of the controls and have every sensor map this to their local definition. For instance, 1 µs for exposure, 1/1000 for gain, etc, and have the max value implicit. DONE

[3]: ~~The "default" value is an arbitrary value of 1 instead of querying the controls of each sensor.~~ DONE

[4]: ~~There is no device const struct uvc_conf and only struct uvc_data, which wastes precious memory.~~ DONE.

[5]: There are missing implementations for selector unit, extensions unit, encoding unit. TODO.

[6]: ~~UVC introduces dependencies between some controls, i.e. auto-expopsure needs to be off for exposure to be accepted.~~ DONE (as part of Zephyr Video).

[7]: ~~UVC has "error" control type reporting the last error.~~ DONE

[8]: Only a single endpoint and output/streaming interface is supported per UVC interface. Composite devices (multiple UVC classes per device) are used instead for supporting multiple video streams per device. DONE

[9]: ~~No documentation outside the devicetree bindings~~. DONE (no documentation needed anymore as no configuration required).

[10]: Supports custom header size, but not passing custom header data yet.

[11]: Still image capture (capturing one frame at full resolution) not supported.

[12]: USB3CV compliance tests not all passing. TODO.

[13]: Announcing different resolutions/FPS for different connection speed not supported.

[14]: Asynchronous controls (the host setting a control, and a notification interrupt alert of completion) not supported.

Supported features:

[A]: Class API and enumeration

[B]: ~~Custom control chains descriptors built from the devicetree layout~~ Not supported by Linux

[C]: ~~Selection of which controls to expose to the host via devicetree toggles~~ Now controlled by the drivers.

[D]: ~~Per-control entity tuning~~ Sending all the controls requests to the first driver of the pipeline.

[E]: Handling of control commands end-to-end from host to Zephyr video device

[F]: Zephyr native Video API that allows to enqueue/dequeue frames like any other Zephyr video device.

[G]: Supports fragmented frames as discussed in #66994 and #72827

[H]: ~~Can configure pixel format and resolution on the devicetree~~ Done automatically by asking the video drivers.

[I]: Supports querying the min/max/current values of every controls end-to-end from host to Zephyr video driver.

subsys/usb/device_next/class/usbd_uvc.c

josuah · 2025-06-26T11:58:23Z

I did not get USB/IP to work so far:

$ west build -b native_sim/native/64 --snippet "usbip-native-sim video-sw-generator" samples/subsys/usb/uvc/
$ west flash
-- west flash: rebuilding
[2/2] Running utility command for native_runner_executable
-- west flash: using runner native
WARNING: Using a test - not safe - entropy source
*** Booting Zephyr OS build v4.1.0-6399-g886a9562b656 ***
[00:00:00.000,000] <inf> net_config: Initializing network
[00:00:00.000,000] <inf> net_config: IPv4 address: 192.0.2.1
[00:00:00.000,000] <inf> uvc_sample: Waiting the host to select the video format

# usbip list -r 192.0.2.1
Exportable USB devices
======================
 - 192.0.2.1
        1-1: NordicSemiconductor : unknown product (2fe3:0011)
           : /sys/bus/usb/devices/usb1/1-1
           : Miscellaneous Device / ? / Interface Association (ef/02/01)
           :  0 - Video / Video Control / unknown protocol (0e/01/00)
           :  1 - Video / Video Streaming / unknown protocol (0e/02/00)

# usbip attach -r 192.0.2.1 -b 1-1

[00:00:22.601,001] <err> usbd_ch9: HWINFO not implemented or enabled
[00:00:22.651,001] <err> usbd_ch9: HWINFO not implemented or enabled
[00:00:22.701,001] <err> usbd_ch9: HWINFO not implemented or enabled
[00:00:22.751,001] <err> usbd_ch9: HWINFO not implemented or enabled
[00:00:22.801,001] <err> usbd_ch9: HWINFO not implemented or enabled
[00:00:22.851,001] <err> usbd_ch9: HWINFO not implemented or enabled

However, it could be detected on Linux to some extent:

# dmesg -c
[ 1078.039367] usb 3-1: New USB device found, idVendor=2fe3, idProduct=0011, bcdDevice= 4.01
[ 1078.039390] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1078.039398] usb 3-1: Product: UVC sample
[ 1078.039405] usb 3-1: Manufacturer: Zephyr Project
[ 1078.188671] uvcvideo 3-1:1.1: Unknown video format 33424752-0000-0010-8000-00aa00389b71
[ 1078.188689] uvcvideo 3-1:1.1: Unknown video format 34325842-0000-0010-8000-00aa00389b71
[ 1078.188715] usb 3-1: Found UVC 1.50 device UVC sample (2fe3:0011)
[ 1078.389396] uvcvideo 3-1:1.1: Failed to set UVC probe control : 0 (exp. 48).

jfischer-no · 2025-06-26T12:15:56Z

I did not get USB/IP to work so far:

I was too hasty, connected too many USB UVC devices to my host.

[ 1078.389396] uvcvideo 3-1:1.1: Failed to set UVC probe control : 0 (exp. 48).

I see it on my side too.

josuah · 2025-06-26T12:28:47Z

Force-push:

CI fix due to missing cast between (uint8_t *) of buf->data and (struct uvc_probe *)'
Use 0-length buffers (correct behavior) instead of empty string "" providing a \0 (hackish).

jfischer-no

@kartben please review

include/zephyr/usb/class/usbd_uvc.h

samples/subsys/usb/uvc/prj.conf

avolmat-st · 2025-06-26T18:33:11Z

samples/subsys/usb/uvc/prj.conf

+CONFIG_LOG=y
+CONFIG_POLL=y
+CONFIG_VIDEO=y
+CONFIG_VIDEO_LOG_LEVEL_WRN=y


Maybe not necessary here ?

Good point, removed to let the default log level apply.

samples/subsys/usb/uvc/prj.conf

kartben

Very neat -- please check my comments, hopefully all pretty straightforward. Thanks!

dts/bindings/usb/zephyr,uvc-device.yaml

samples/subsys/usb/uvc/README.rst

subsys/usb/device_next/class/usbd_uvc.c

samples/subsys/usb/uvc/README.rst

include/zephyr/usb/class/usbd_uvc.h

samples/subsys/usb/uvc/README.rst

josuah · 2025-06-26T19:51:39Z

force-push:

Let default log level apply
Let Kconfig defaults apply for CONFIG_VIDEO_BUFFER_POOL_NUM_MAX and CONFIG_VIDEO_BUFFER_POOL_SZ_MAX.
Add documentation of the public usbd_uvc.h header API
Readability adjustments

include/zephyr/usb/class/usbd_uvc.h

Introduce a new USB Video Class (UVC) implementation from scratch. It exposes a native Zephyr Video driver interface, allowing to call the video_enqueue()/video_dequeue() interface. It will query the attached video device to learn about the video capabilities, and use this to configure the USB descriptors. At runtime, this UVC implementation will send this device all the control requests, which it will send to the attached video device. The application can poll the format currently selected by the host, but will not be alerted when the host configures a new format, as there is no video.h API for it yet. Signed-off-by: Josuah Demangeon <me@josuah.net>

kartben · 2025-06-26T21:39:18Z

samples/subsys/usb/uvc/README.rst

@@ -0,0 +1,183 @@
+.. zephyr:code-sample:: uvc
+   :name: USB Video webcam
+   :relevant-api: usbd_api video_interface


No need to fix now (I mean it!) but in case there is another round of updates, just realized we now have a proper doxygen group for the UVC device api

Suggested change

:relevant-api: usbd_api video_interface

:relevant-api: usbd_api usbd_uvc video_interface

Good catch! I will aggregate all the non-blocking fixes to this branch then...
https://github.com/tinyvision-ai-inc/zephyr/pull/new/pr_fix_uvc

josuah · 2025-06-26T22:38:51Z

We still have all the non-video boards working without problem. For instance:

west build -b rpi_pico --snippet video-sw-generator samples/subsys/usb/uvc/

west build -b nrf52840dongle --snippet video-sw-generator samples/subsys/usb/uvc/ -- \
  -DCONFIG_USBD_DEVICE_STACK=n -DCONFIG_USBD_DEVICE_STACK_NEXT=y # I need to get a real nrf52840dk!

west build -b frdm_mcxn947/mcxn947/cpu0 --snippet video-sw-generator samples/subsys/usb/uvc/

jfischer-no · 2025-06-27T07:55:08Z

@josuah The logging settings that were removed here should be brought back. I am also confused why they were removed?

Following the addition of USB Video Class, this adds a sample that makes use of the zephyr,camera chosen node of any board to stream the video source to the host. A snippet video-sw-generator can be used to test and debug devices without a zephyr,camera chosen node. Signed-off-by: Josuah Demangeon <me@josuah.net>

josuah · 2025-06-27T09:08:58Z

@josuah The logging settings that were removed here should be brought back. I am also confused why they were removed?

I assumed the default log level was WRN and that the configuration was redundant, but no it is INF so removing this changed the behavior. I added them back like asked here.

sonarqubecloud · 2025-06-27T09:34:04Z

Quality Gate passed

Issues
16 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud

jfischer-no

@josuah Thank you very much. This is a significant and valuable contribution to the USB device subsystem. Thank you also for your patience.

josuah · 2025-06-27T10:29:09Z

👍

The time spent waiting allowed to introduce UVC directly on top of the latest video and USB APIs, and testing it plenty.

		.dwMinBitRate = sys_cpu_to_le32(15360000), \
		.dwMaxBitRate = sys_cpu_to_le32(15360000), \

	:relevant-api: usbd_api video_interface
	:relevant-api: usbd_api usbd_uvc video_interface

usb: device_next: new USB Video Class (UVC) implementation #76798

usb: device_next: new USB Video Class (UVC) implementation #76798

Conversation

josuah commented Aug 7, 2024 • edited by kartben Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josuah commented Aug 7, 2024

Uh oh!

josuah commented Aug 8, 2024

Uh oh!

josuah commented Aug 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

josuah commented Aug 8, 2024

Uh oh!

josuah commented Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josuah commented Sep 7, 2024

Uh oh!

josuah commented Sep 7, 2024

Uh oh!

josuah commented Sep 12, 2024

Uh oh!

josuah commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Known limitations:

Supported features:

Uh oh!

Uh oh!

josuah commented Jun 26, 2025

Uh oh!

jfischer-no commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josuah commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfischer-no left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kartben left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

josuah commented Jun 26, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josuah commented Aug 7, 2024 •

edited by kartben

Loading

josuah commented Aug 21, 2024 •

edited

Loading

josuah commented Sep 12, 2024 •

edited

Loading

jfischer-no commented Jun 26, 2025 •

edited

Loading

josuah commented Jun 26, 2025 •

edited

Loading

jfischer-no left a comment •

edited

Loading

josuah commented Jun 26, 2025 •

edited

Loading

josuah commented Jun 27, 2025 •

edited

Loading